THE DATA CRUNCH 



Harnessing the 

POWEF 


BY SUSAN REESE 


T he amount of and demand 
for high-quality, accessible 
education data are increas- 
ing, according to the Data 
Quality Campaign, which is managed 
by the National Center for Educational 
Accountability and supported by the Bill 
and Melinda Gates Foundation. Some of 
this increase is due to new technology that 
has made the collection and analysis of 
data easier and more efficient, and some 
is due to the demands for greater account- 
ability, such as required by No Child Left 
Behind. There are also growing numbers 
of stakeholders both within and outside 
the educational system calling for better 
information about our children’s educa- 
tional experience, such as how it prepares 
them for future careers and postsecond- 
ary education. 

Data can be used to document needs 
for improvement and as a tool for reform, 
but it can also be used to demonstrate 
effectiveness. While many career and 
technical education (CTE) programs 
have anecdotes to share about the suc- 
cess of their graduates, increasingly 
there is a call for more than anecdotal 
evidence. The demand is for hard, cold 
facts — data to back up the need for, and 
success of, career-tech. Richard Lynch, 
professor emeritus and former director 


of the School of Leadership and Lifelong 
Learning at the University of Georgia, 
notes, “It is often estimated that less than 
one-fourth of one percent of education 
budgets are ever spent on research and 
innovation. Contrast this with five to 15 
percent often cited in most industry 
sectors such as engineering, medicine, 
health care and business. Nearly all 
serious education policy groups and 
reformers at local, state and national 
levels call for more substantive infusion 
of funds for education research and evalu- 
ation studies.” 

Lynch devoted a year of his life to 
collecting data and writing his paper, New 
Directions for High School Career and Technical 
Education. When asked why he was willing 
to make such a large commitment, Lynch 
provides this answer. “The assistant secre- 
tary of education at the time asked me to 


spend the year in D.C. and write a ‘vision’ 
or ‘direction’ paper for high school ‘voca- 
tional education.’ This was in response to 
the policy-level question she was constant- 
ly being asked: ‘What is (or should be) the 
contemporary role, purposes, structure 
and value of high school CTE?’ I think, in 
effect, so many thought high school CTE 
should be eliminated. Trish McNeil, the 
assistant secretary, didn’t want to do so 
without, well. . .., data or good informa- 
tion or perspective.” So data can be a 
pretty powerful tool, and sometimes that 
power can be harnessed and used to make 
a strong case for CTE. 


Outside Sources 

There are two main ways that schools and 
districts are utilizing data in making the 
case for CTE. The first is to use infor- 
mation that has been gathered by other 
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R E FO R M, but it can also be used to demonstrate EFFECTIVENESS. 


organizations. The search should begin 
with our own organization’s Web site. 

On the CTE Information and Research 
page ofwww.acteonline.org, there is 
data and research “that supports better 
understanding of today’s CTE systems, 
programs and services, and highlights 
the potential that CTE provides students 
related to technical skills, academic 
achievement and career guidance.” 

The ACTE site offers a CTE Effective- 
ness Fact Sheet, a collection of promising 
programs and practices, and a link to the 
CTE Research Clearinghouse. ACTE 
is the dissemination arm of the National 
Research Center for Career and Techni- 
cal Education (www.nrccte.org), an 
excellent source for data on best practices 
in CTE. 

Some of the benefits of CTE have been 
recognized by organizations such as the 
National Dropout Prevention Center, 
which cites career academies, tech prep, 
cooperative education, school-based 
enterprises, internships and apprentice- 
ships, job shadowing, mentoring and 
career guidance as strategies used for 
keeping students in school. The Center 
for Occupational Research and Develop- 
ment is a source for positive information 
about tech prep, and the smaller-school 
initiatives of the Bill and Melinda Gates 
Foundation often provide evidence to sup- 
port the benefits of the career academy 
model. The Southern Regional Educa- 
tion Board has also gathered data on best 
practices and the benefits of high-quality 
CTE programs. 

Lynch is among those who say that 
at this point in time, it is better not to 
mandate collecting more data from teach- 


ers, especially if the sole purpose seems 
to be for state or federal report purposes, 
but rather, we should use the data that we 
already collect from and about students 
and teachers to help improve CTE where 
warranted. 

“I often chuckle at the quip by 
Jeffrey Wayman, a researcher from Johns 
Hopkins Center for Social Organization 
of Schools,” says Lynch of the com- 
ments that appeared in the December 14, 
2005, issue of Education Week. “Wayman 
said that data about students too often 
‘languish in central repositories used for 
little but accountability reporting.’ More 
succinctly, ‘Data have been like a roach 
motel. . . [they] check in, they just don’t 
check out.’” 

Lynch believes that the multitude of 
quantifiable data ought to be tabulated 
and analyzed at the local level either 
before and/or in tandem with it being 
sent off to the next level. “This refers 
mostly to number crunching at the 
individual and class (or program) level: 
student enrollments, demographics on 
students, programs of study, standardized 
test scores, socio-economic status, atten- 
dance record, subjects completed, grades, 
teacher qualifications, and on and on,” 
he explains. “The data currently being 
collected and in student hies can provide 
to teachers and administrators objective 
information to help plan instruction 
based on identified and objectively 
analyzed data.” 

Lynch adds that these data can also 
be correlated with data that are collected 
concomitantly or as a follow-up sometime 
at a later point in students’ education or 
employment. “For example, although 


challenging often due to turf protection, 
high school student-based data can be 
bumped up against labor market data 
and/or postsecondary agency data being 
collected simultaneously or later,” he 
notes. “In Georgia, this is how we were 
able to follow up and determine outcomes 
of dual enrollment on students’ employ- 
ment and postsecondary education.” 

Internal Sources 

The Oklahoma Department of Career 
and Technology Education is often cited 
as a model for statewide CTE, and not 
only does the state department have posi- 
tive information to share, but some of the 
technology centers themselves collect and 
disseminate data about their success. 

Metro Technology Centers issues an 
annual Progress Report to Stakeholders 
listing its organizational strategies and 
how they have been addressed. In its 2007 
report, Metro Tech cites data such as the 
pass rates of its students for licensures/ 
certification competency tests, which 
included 100 percent passing rates for 
five areas, with the remaining three areas 
having passing rates of 96 percent, 95 
percent and 86 percent. The report docu- 
ments positive student placement rates. 
The data also showed five-year increases 
in the number of students participating 
in cooperative agreements, college hours 
earned through cooperative agreements 
and student completion/retention rates. 
The total enrollment at Metro Tech has 
grown by 109 percent over a six-year 
period, but there has also been growth in 
another important area — student satisfac- 
tion. Student surveys as well as surveys 
of the school’s advisory committee found 
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high rates of satisfaction among both stu- 
dents and business and industry partners. 

“Metro Technology Centers’ dramatic 
improvement in key measures (comple- 
tion/retention, positive placement and 
licensure/pass rates) is directly attribut- 
able to our future picture and desired 
effects,” says Metro Tech Superintendent 
James Branscum. “Our goal is continuous 
improvement in all result categories. This 
requires ‘liquid planning’ that allows 
strategy to change as does our global 
society — quickly. MTC uses the Baldrige 
business model to plan strategy because it 
is non-prescriptive, customer focused and 
results oriented.” 

Another Oklahoma Technology 
Center, Francis Tuttle, issues an annual 
report, which notes that, over the last 
five years, Francis Tuttle has averaged a 
94.7 percent positive student placement, 
defined as employment in the career area 
of training, service in the military, con- 
tinuing education, and/or employment 
in another held. According to the Francis 
Tuttle 2007 Annual Report, 78.7 percent 
of high school students who completed a 
program at Francis Tuttle were continu- 
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ing their education in 2007. 

“Collecting and analyzing data paves 
the way for effective decision-making 
and helps to assure the taxpaying public 
that we are serving as good stewards of 
their funds and making decisions in their 
best interests,” says Francis Tuttle CEO/ 
Superintendent Kay Martin. “We have 
always encouraged students to continue 
their education, as data indicates that all 
education enhances earnings potential, 
aside from benefiting their families and 
the communities in which they live. We 
have also found that many students come 
to us following completion of four-year 
(and advanced) degrees.” 

Wisconsin’s Gateway Technical 
College (GTC) has learned about the 
power of data and has made very effective 
use of the positive statistics it has gath- 
ered. For example, the 2007 Graduate 
Follow-up Study found that 95 percent of 
graduates are satisfied with their Gateway 
training, 92 percent found jobs within six 
months of graduation, and 67 percent are 
employed in their held of training. 

“It is critical to use program and 
student data to engage community sup- 


"Collecting and analyzing 
data paves the way for 
EFFECTIVE 
DECISION- 
MAKING and helps to 
assure the taxpaying public that 
we are serving as good stewards 
of their funds and making 
decisions in their best interests/' 
says Francis Tuttle CEO/ 
Superintendent Kay Martin. 


port and build common understanding,” 
says Gateway president — and current 
ACTE president — Bryan Albrecht. 

GTC serves the counties of Kenosha, 
Racine and Walworth, providing educa- 
tion, training, leadership and technologi- 
cal resources to ensure economic growth 
and meet the changing needs of the area’s 
students, employers and communities. It is 
providing those services very effectively — 
and has the data to prove it. The Gateway 
Technical College Foundation commis- 
sioned a study by CCbenehts, a company 
that provides economic impact analysis 
on the benefits generated by community 
and technical colleges in the United States 
and Canada. The study tracks four types 
of benefits: regional economic benefits 
(local job and income formation); student 
perspective (higher earnings captured by 
exiting students); taxpayer perspective (a 
collection of social benefits and avoided 
costs); and investment analysis (the return 
to taxpayers for their college support). 

The study, “The Economic Contri- 
bution of Gateway Technical College,” 
found that, “The GTC Service Area 
economy owes roughly $401.4 million of 
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its regional income to GTC operations 
and past student productivity effects.” 
Now that is a very impressive figure and 
certainly powerful enough to make 
students, employers and community 
leaders take notice. According to the 
study, “For every credit completed, GTC 
students will, on average, earn $124 
more per year each year they are in the 
workforce. Alternatively, for every full- 
time year they attend they will earn an 
additional $3,721 per year.” This makes 
GTC a very good investment for students, 
who will earn an 18 percent rate of return 
on their investments of time and money. 

State and local government spent 
$49.5 million in support of GTC during 
the year that was analyzed by the study, 
but the data shows that this was a good 
investment for taxpayers. In addition 
to their increased earnings cited above, 
those with higher education are less likely 
to smoke, abuse alcohol, draw welfare 
or unemployment benefits, or commit 
crimes. “This translates into associated 
dollar savings (avoided costs) amounting 
to some $17 per credit per year, counted 
as an indirect benefit of GTC education,” 
notes the study. “When aggregated across 
all exiting students, the State of Wiscon- 
sin will benefit from $2.1 million worth of 
avoided costs per year, each year that the 
students are in the workforce.” 

All in all, the report presents a power- 
ful case for the benefits of CTE program- 
ming provided by the college. 

As Albrecht notes — and his statements 
are supported by the data collected by 
his school — “The impact of CTE has 
multiple dimensions, including traditional 
measures such as academic achievement, 
retention and graduation rates, and 
student interest and engagement, as well 
as broad community growth measures 
like increased employment opportunities, 
increased personal income and having the 
ability to add value to personal property 
by CTE graduates, and reduced social 
services costs associated with under or 
unemployment rates in a community. 
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“WITHOUT VALID AND RELIABLE DATA 

we are simply flying by the seat of our pants, and no serious, 
objective-minded policy wonk is going to pay any attention to our 
pleas for support." — Professor Richard Lynch. 


The value of GTE is best demonstrated 
by comprehensive community impact 
measures in addition to the highly valued 
educational attainment measures we are 
accustomed to.” 

Using Data Wisely 

Lynch advises that, “Quantifiable data, 
important as it may be, should never 
replace the wisdom and judgment of 
teachers, other practitioners and parents. 
Good teachers know students well and 
know how students learn; their collec- 
tive wisdom can and should be used in 
powerful ways to improve and strengthen 
CTE.” 

Lynch also believes that CTE needs 
to take the lead to expand the conversa- 
tions with policy groups, legislators and 
education reform advocates about what it 
means to achieve. “Student achievement 
is far, far more than a score on a norm- 
or even criterion-referenced standardized 
test,” he adds. “Further policy edicts just 
must include the measurement of student 
achievement through successful assess- 
ment of student work on age-appropriate, 
complex projects, occupational certifi- 
cations earned, state licenses awarded, 
and other demonstrable measures of 
competence. Other factors might also 
include attendance, graduation, transition 
into employment and/or postsecondary 
education, scores on standardized tests 
that measure real-world skills, assessment 
of soft skills, and student organization 
competitions. Overall, I think it only just 
and fair to showcase student achieve- 


ment through a portfolio of student work 
and accomplishments, and yes, this can 
include scores on standardized tests of 
academic achievement.” 

Over the years, Lynch has noted that 
many fine CTE administrators and 
teachers are cautious or defensive about 
disseminating any data, especially of the 
accountability type that state and federal 
agencies have called for, and he thinks 
that this may be the cause of data tend- 
ing to languish in those repositories as 
noted by Wayman. “I think such fear or 
hesitancy of releasing data is unfortunate, 
as it is pretty difficult to improve if the 
data aren’t known or transparent,” he 
adds. Instead he would encourage those 
who have concerns about disseminating 
such data to try to better understand what 
is being asked of the data and, hopefully, 
be involved in setting the objectives of the 
data collection. 

“Conceptually, it is pretty simple,” 
Lynch explains. “What is it you need to 
know? How will you come to know it? 
How will you know what you got? And 
then use the data accordingly; that is, to 
either advocate for more and improved 
CTE and/or to make the modifications 
that are warranted.” 

He sums it up this way: “Without valid 
and reliable data — including that trian- 
gulated through qualitative measures and 
the wisdom of teachers — we are simply 
flying by the seat of our pants, and no 
serious, objective-minded policy wonk is 
going to pay any attention to our pleas for 
support.” D 


Exploring Sources of Data \ 

For more information about harnessing \ 
the power of data to make a case for the \ 
benefits of career and technical education, 
here are some sources to explore. 

Association for Career and Technical 
Education (www.acteonline.org) 

Information and Research (www.acteonline. 
org/content.aspx?id= 206 ) 

National Research Center for Career 
and Technical Education 

www.nrccte.org 

Center for Occupational Research 
and Development 

www.cord.org 

National Dropout Prevention Center 

www.dropoutprevention.org 

The Bill and Melinda Gates Foundation 

www.gatesfoundation.org 

Data Quality Campaign 

www.dataqualitycampaign.org 

Southern Regional Education Board 

www.sreb.org 

About the Schools 

To learn more about the schools featured 
in this story, visit their Web sites. 

Gateway Technical College 

www.gtc.edu 

Metro Technology Centers 

www. m etrotec h . o rg 

Francis Tuttle Technology Center 

www.francistuttle.com 


Susan Reese 


is a Techniques contributing writer. She can be contacted at 
susan@printmanagementinc.com. 

Interested in exploring this topic 
further? Discuss it with your 
colleagues on the ACTE forums at www. 
acteonline.org/forum.aspx. 
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ft Perkins challenge: 

Assessing Technical skills in CTE 



BY JAMES R. STONE III 

C urrent federal law requires states to develop 
performance measures and data collection 
systems for the four required core indicators, 
including technical skill attainment for sec- 
ondary and postsecondary levels. These measures must be 
reliable and valid. Perkins IV requires states to use state- 
established, industry-validated career and technical skill 
standards; but few of these exist and states have largely been 
compelled to rely on career and technical education (GTE) 
course completion, grade point averages (GPA) in CTE 
courses, or administrative data. 

Challenges 

The first challenge posed by the Perkins requirement is 
defining a technical skill. How do we parse technical skills 
from non-technical skills? The most common approach 
relates technical skills to skill standards which are “perfor- 
mance objectives and competencies required by a specific 
occupation, as specified by experts within that particular in- 
dustry” (Center for Remediation Design, 1991), and “perfor- 
mance specifications that identify the knowledge, skills and 
abilities an individual needs to succeed in the workplace.” 
According to the National Skill Standards Board, skill stan- 
dards “. . .consist of two components: (1) a description of the 
responsibilities needed for competent performance, and (2) 
a description of knowledge and skills necessary to carry out 
these responsibilities” (2000). 

A second challenge is measurement. True performance 
assessments, sometimes known as authentic assessment, 
require a student to perform. Performing a task, the process 
of doing, rather than selecting an answer from a ready-made 
list (circle “D” for “none of the below”) is a hallmark of per- 
formance assessments. The U.S. Congress Office of Tech- 
nology Assessment defines performance assessment as “any 
form of testing that requires a student to create an answer or 
a product that demonstrates his or her knowledge or skills.” 
This requires experienced raters who judge the quality of the 
student’s performance against established criteria; consider 
driver certification tests or Olympic events. Closer to home, 
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